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Abstract 

The  limitations  of  rigid  fingertips  in  the  precise  and 
algorithmic  study  of  manipulation  have  been  discussed 
in  many  works,  some  dating  back  more  than  a  decade. 
Despite  that  fact,  much  of  the  work  in  dexterous  ma¬ 
nipulation  has  continued  to  use  the  “point-contact” 
model  for  finger-object  interactions.  In  fact,  most  of 
the  exsisting  tactile  sensing  technologies  are  not  adapt¬ 
able  to  deformable  fingertips.  In  this  work  we  report  on 
experimental  results  obtained  with  a  deformable  tactile 
sensor  whose  properties  are  well-suited  to  manipula¬ 
tion.  The  results  presented  here  show  that  the  sensor 
described  provides  a  rich  set  of  tactile  data. 


1  Introduction 

In  this  work  we  describe  a  deformable  image-based  tac¬ 
tile  sensor  whose  output  is  an  approximation  of  the 
tactile  surface  itself.  We  present  a  set  of  basic  tactile 
sensing  experiments  designed  to  demonstrate  aspects 
of  the  sensor’s  performance.  The  ability  of  our  sensor 
to  deform  while  accurately  localizing  contact  (s)  makes 
it  a  promising  tool  for  use  in  dexterous  manipulation 
and  other  applications. 

Tactile  sensors  have  been  widely  used  in  manipula¬ 
tion  tasks  [19],  edge  following  [2],  as  well  as  automatic 
grasping  [3]  and  compliance  control  [7].  One  of  the 
disadvantages  of  conventional  tactile  sensors  is  that 
they  are  inherently  two-dimensional,  meaning  that  they 
sense  a  pressure  distribution  over  their  surface  but  pro¬ 
vide  little  or  no  information  on  possible  deformations  of 

1  This  work  was  funded  by  the  followging  grants:  Army  DAAG 
55  97  0114,  NSF  EEC  94  02384  and  Army  DAAL  03-92-  G  0115. 

2  This  work  was  done  while  the  author  was  with  the  Divi¬ 
sion  of  Engineering  and  Applied  Sciences  at  Harvard  University, 
Cambridge,  MA. 


the  surface  itself.  With  few  exceptions  ([22],  [24]),  tac¬ 
tile  arrays  are  typically  mounted  against  a  rigid  back¬ 
ing  and  covered  with  a  thin  rubber  layer  to  provide 
friction.  This  rigidity  limits  the  degree  to  which  such 
sensors  can  be  used  in  the  study  of  manipulation  tasks 
[5] .  In  addition,  most  of  the  existing  tactile  arrays  are 
flat,  so  they  must  be  mounted  on  flat  or  cylindrical 
fingertips.  Shimoga  and  Goldenberg  explored  differ¬ 
ent  ways  of  constructing  non-rigid  fingertips  [26,  27]. 
Various  materials  including  foam,  rubber,  powder  and 
gel  were  investigated.  The  gel-filled  membrane  showed 
best  overall  performance  in  terms  of  attenuation  of  im¬ 
pact  forces,  conformability,  strain  dissipation,  and  re¬ 
ality  factors.  The  compliant  fingertips  used  this  paper 
most  closely  resemble  the  gel-filled  fingertip  used  by 
Shimoga. 

Over  the  last  two  decades,  tactile  sensing  research  has 
focused  on  the  development  of  technology  and  devices 
that  attempt  to  endow  robots  with  some  of  the  dex¬ 
terity  that  humans  possess.  Everyday  experience  as 
well  as  analysis  of  the  kinematics  of  manipulation  and 
grasping  [17,  21]  suggest  that  contact  forces  and  loca¬ 
tions  are  the  most  important  geometric  parameters  for 
manipulation  and  it  is  precisely  those  parameters  that 
most  tactile  sensors  are  designed  to  measure.  We  will 
briefly  state  some  of  the  most  important  examples  of 
tactile  sensing  technologies,  as  they  pertain  to  manip¬ 
ulation.  See  [14]  for  a  comprehensive  review. 

Many  of  the  tactile  sensors  in  existence  use  piezo- 
resistive  [12,  28]  or  capacitive  arrays  [9,  14]  of  pressure- 
sensitive  elements  which  when  in  contact  with  an  ob¬ 
ject,  can  provide  information  on  the  location  of  the 
contact  and  its  pressure  distribution.  Capacitive  tac¬ 
tile  arrays  are  usually  preferred  over  their  piezo-electric 
counterparts  due  to  their  higher  sensitivity,  linear  re¬ 
sponse  and  lack  of  hysteresis.  Optical  tactile  sensors 
are  an  alternative  to  tactile  arrays  and  been  devel¬ 
oped  for  contact  localization,  including  [23],  [1]  and 
[20].  Other  image- based  tactile  sensors  are  described 
in  [6]  and  [10].  In  [25],  [4]  and  [8]  “intrinsic”  tactile 
sensing  is  explored,  where  knowledge  of  a  (rigid)  fin¬ 
gertip’s  shape  is  used  in  conjunction  with  force-torque 


sensors.  A  significant  amount  of  work  has  involved  dy¬ 
namic  tactile  sensing.  Dynamic  tactile  sensors  (e.g. 
[15,  13,  18])  typically  measure  transient  contact  effects 
such  as  vibrations,  stress  changes  and  slip. 


2  A  3-D  Deformable  Tactile  Sensor 

Figure  1  shows  the  deformable  tactile  sensor  that  has 
been  developed  in  the  Harvard  Robotics  Lab  as  a  re¬ 
sult  of  a  decade-long  collaborative  effort.  A  complete 
description  of  the  sensor  and  its  operation  can  be  found 
in  [10].  The  sensor  consists  of  a  metal  housing  and  a 
roughly  elliptical  latex  membrane  which  provides  an 
area  of  contact.  A  clear,  fluid-like  gel  fills  the  mem- 


Figure  1:  The  tactile  sensor. 

brane,  sealed  from  the  rest  of  the  assembly  by  a  trans¬ 
parent  window.  A  grid  of  dots  is  drawn  at  precisely 
computed  locations  on  the  inner  surface  of  the  mem¬ 
brane.  A  metal  fingernail  serves  to  provide  support  for 
the  membrane  when  it  is  being  deformed  by  contact. 
The  fingertip  is  approximately  6.2 an  long  and  has  a 
diameter  of  2 cm  at  its  base.  A  schematic  is  shown  in 
Fig.  2.  The  sensor’s  metal  housing  holds  a  camera  with 


Figure  2:  Tactile  sensor  schematic. 

a  diameter  of  7.5 mm  and  a  fiber  optic  cable  that  illumi¬ 
nates  the  interior  surface  of  the  membrane.  The  camera 
is  connected  to  an  image  acquisition  board  which  cap¬ 
tures  images  of  the  grid  of  dots  on  the  membrane.  Typ¬ 
ical  images  are  shown  in  Fig.  3.  The  image  size  that  was 


used  was  192  x  120  pixels.  The  sensor  has  mechanical 


Figure  3:  Camera  view  of  membrane:  (a)  undeformed  (b) 
in  contact  with  an  object. 

properties  that  are  much  better  suited  to  manipulation 
than  those  of  conventional  robotic  sensors.  In  particu¬ 
lar,  the  use  of  a  fluid-supported  membrane  ([5])  allows 
local  deformations  (caused  by  contact  with  an  object) 
to  be  distributed  throughout  the  enclosed  volume,  be¬ 
cause  of  the  constant  pressure  of  the  fluid  inside.  This 
is  in  contrast  to  materials  that  obey  Hooke’s  law  (i.e. 
rubber-covered  rigid  fingertips)  and  allows  the  finger- 
pad  to  locally  “wrap  around”  the  object  at  a  contact. 
Mechanically,  the  sensor  acts  much  like  a  human  fin¬ 
gertip  (albeit  more  compliant)  and  is  very  effective  in 
providing  grasp  stability. 


3  Membrane  Shape  Reconstruction 

The  locations  of  the  dots  on  the  membrane  are  known 
a  priori.  When  the  fingertip  comes  in  contact  with  the 
environment,  the  membrane  deforms  and  the  camera 
observes  a  change  in  the  projections  of  the  grid  of  dots 
onto  the  image  plane  (as  in  Fig.  3-b).  Projective  geom¬ 
etry  tells  us  that  there  exist  an  infinity  of  solutions  for 
the  new  three-dimensional  coordinates  of  the  dots.  Un¬ 
der  deformation,  the  portion  of  the  membrane  which  is 
not  in  contact  will  assume  a  shape  that  minimizes  its 
elastic  energy.  In  addition,  the  volume  enclosed  by  the 
membrane  remains  constant,  and  the  boundary  of  the 
membrane  is  fixed.  These  constraints,  together  with 
some  genericity  assumptions  on  the  grid  of  dots  are 
sufficient  to  obtain  a  solution  for  the  three-dimensional 
coordinates  of  the  grid.  The  algorithm  used  to  accom¬ 
plish  this  (termed  “the  reconstruction  algorithm”)  is 
presented  in  [10,  11]. 

Briefly,  the  reconstruction  algorithm  uses  images  such 
as  the  one  in  Fig.  3  to  produce  a  three-dimensional  ap¬ 
proximation  of  the  membrane  surface,  in  the  form  of 
a  13  x  13  mesh  that  corresponds  to  a  4c?n2  area  on 
the  fingerpad.  The  steps  of  the  membrane  reconstruc¬ 
tion  algorithm  are  depicted  in  Fig.  4.  Details  of  the 
algorithm  are  presented  elsewhere  [11].  A  reconstruc¬ 
tion  example  is  shown  in  Fig.  5,  corresponding  to  a 
human  fingertip  lightly  touching  the  membrane.  The 
coordinates  of  the  grid  are  measured  with  respect  to  an 
inertial  frame  whose  origin  at  the  center  of  the  CCD 
array  in  the  camera  and  whose  z-axis  is  perpendicular 


to  that  array.  “Crossed”  points  represent  the  unde¬ 
formed  location  of  the  grid.  The  straight  line  through 
the  grid  is  drawn  through  the  centroid  of  the  area  of 
contact  (see  Sec.  4).  The  reconstruction  algorithm  as¬ 
sumes  that  membrane  deformations  are  “small” . 


Figure  4:  The  fingertip  operation:  (a)  a  pattern  of  dots 
is  drawn  on  the  interior  of  a  fluid-filled  mem¬ 
brane.  (b)  The  membrane  deforms  when  in 
contact  with  objects,  (c)  Image  data  of  the 
displacement  of  the  pattern  of  dots  is  used  to 
interpolate  a  flow  field,  (d).  The  image  flow 
field,  along  with  other  constraints  enable  re¬ 
construction  of  the  3D  shape  of  the  deformed 
membrane  (e). 


In  addition,  the  algorithm  involves  a  significant  amount 
of  computation  and  image  processing.  On  a  dual 
400 MHz  Pentium  PC  the  maximum  rate  of  perform¬ 
ing  this  reconstruction  is  15 Hz  using  a  5  x  5  grid  of  dots 
on  the  membrane  and  a  13  x  13  interpolated  grid  to  ap¬ 
proximate  the  fingerpad  surface.  This  rate  is  low  com¬ 
pared  to  those  that  can  be  achieved  with  traditional 
tactile  sensors,  however  the  deformable  sensor  provides 
a  much  richer  description  of  a  contact.  Using  denser 
grids  for  the  membrane  surface  increases  the  precision 
of  the  tactile  data  as  well  as  the  computation  time  for 
a  single  reconstruction. 


Figure  5:  Tactile  sensing  example. 


4  Tactile  Sensor  Performance 

In  the  following,  we  describe  a  set  of  tactile  sensing  ex¬ 
periments  that  were  designed  to  demonstrate  the  per¬ 
formance  of  our  tactile  sensor.  In  these  experiments, 
we  were  interested  in  evaluating  the  sensor’s  accuracy 
(in  contact  localization  tasks),  spatial  resolution,  re¬ 
construction  accuracy  and  curvature  discrimination. 

Contact  Localization:  From  the  three- 
dimensional  reconstruction  of  the  fingerpad  we  can 
estimate  which  portion  of  the  membrane  is  in  contact 
with  an  object.  Consider  the  reconstruction  example 
of  Fig.  5.  By  computing  the  displacement  along  the 
inward-pointing  normal  for  each  point  on  the  grid,  we 
can  identify  the  points  which  are  part  of  a  contact. 
Figure  6  shows  typical  results  obtained  with  this 
method  when  a  pencil  tip  is  pressed  lightly  against  the 
fingerpad.  The  graph  shows  a  peak  forming  around 
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Figure  6:  Contact  detection. 


the  area  of  contact  from  which  we  can  determine  that 
the  pencil  was  pressed  about  1  mm  into  the  membrane. 
The  area  of  contact  included  14  grid  points  with  their 
centroid  at  (—4.5 mm,  —3.7 mm,  22.2mm )  measured  in 
a  coordinate  frame  located  at  the  end  of  the  distal 
link.  In  the  following,  we  will  use  the  terms  “contact” 
or  “contact  location”,  to  refer  to  the  centroid  of 
the  area  of  contact.  Our  tactile  sensor  is  able  to 
simultaneously  detect  multiple  areas  of  contact,  as 
shown  in  Fig.  7.  The  minimum  inward  displacement 
that  can  be  detected  is  0.5 mm. 


Z(mm) 


Figure  7:  Double  contact  detection. 


Constant  lighting  conditions  and  sufficient  image  res¬ 
olution  are  necessary  in  order  for  the  reconstruction 
algorithm  to  repeatably  locate  the  projections  of  the 
membrane  dots  on  the  image.  The  following  experi¬ 
ment  was  performed  in  order  to  measure  the  noise  level 
associated  with  detecting  dot  projections  in  the  cam¬ 
era  image:  The  membrane  was  kept  motionless,  100 
images  like  the  one  shown  in  Fig.  3  were  taken  and  the 
centroid  of  each  dot  was  computed.  The  standard  de¬ 
viation  of  the  noise  was  approximately  0.70  pixels  and 
0.61  pixels  in  the  horizontal  and  vertical  directions  re¬ 
spectively.  If  we  identify  a  contact  by  the  centroid  of  all 
grid  points  that  are  part  of  that  contact  then  the  error 
(due  to  lighting  noise)  in  computing  the  coordinates  of 
the  contact  had  a  norm  less  than  0.1»nm. 

Accuracy  under  Small  Deformations:  The 

accuracy  of  the  reconstructed  grid  depends  on  the  spa¬ 
tial  density  of  dots  drawn  on  the  membrane.  Currently 
a  5  x  5  grid  is  used,  covering  an  area  of  4cm2.  In 
order  to  measure  the  accuracy  with  which  the  sensor 
can  localize  contact  over  its  surface  under  small  de¬ 
formations,  the  following  experiment  was  performed. 
The  sensor  was  mounted  on  an  apparatus  (pictured  in 
Fig.  8)  which  allows  an  indenter  to  be  brought  in  con¬ 
tact  with  the  fingerpad.  The  indenter  is  mounted  on  a 
5  degree-of-freedom  assembly  so  that  it  can  always  be 


Figure  8:  Indenter  apparatus. 


oriented  along  the  surface  normal  over  any  location  on 
the  fingerpad.  By  sliding  the  indenter  along  the  surface 
normal  we  can  produce  a  desired  inward  displacement 
of  the  membrane  at  the  contact  location.  This  was 
done  for  a  set  of  25  points  which  were  distributed  over 
the  entire  fingerpad  and  whose  coordinates  had  been 
previously  measured.  The  indenter  used  was  a  metal 
rod  with  a  diameter  of  2.5 mm,  designed  to  approxi¬ 
mate  a  “point”  indenter  without  damaging  the  latex 
membrane.  The  inward  displacement  at  the  contact 
was  always  kept  at  0.5 mm.  We  indented  the  mem¬ 
brane  at  each  of  the  chosen  points  and  obtained  the 
corresponding  reconstructed  grid.  For  each  reconstruc¬ 
tion,  the  centroid  of  the  contact  area  was  computed  in 
order  to  identify  the  contact  location.  Finally,  the  co¬ 
ordinates  of  the  contact  location  were  compared  with 
the  actual  coordinates  of  the  point  on  the  membrane 
that  was  in  contact  with  the  indenter.  The  norm  of 
the  resulting  error  vector  had  a  mean  of  0.75 mm.  The 
maximum  error  was  1.9mm,  equal  to  one  half  of  the 
distance  between  neighboring  dots  on  the  membrane 
surface. 

Deformation  Depth:  Two  different  indenters 
were  used  to  deform  the  membrane  1  mm,  2 mm,  3 mm 
and  4 mm  along  its  surface  normal  at  each  of  the  25  dots 
drawn  on  the  membrane.  In  addition  to  the  “point”  in¬ 
denter,  a  2.54cm-diameter  flat  rigid  disk  was  used  to 
deform  the  membrane  over  a  large  area.  In  each  case 
the  indenter  was  normal  to  the  surface,  held  steady  by 
the  apparatus  used  in  the  previous  experiment.  The 
reconstructed  grid  was  used  to  compute  the  maximum 
inward  displacement  of  the  tactile  surface,  which  was 
then  compared  to  the  actual  displacement  effected  by 
the  indenter.  Tables  1  and  2  show  the  mean  and  stan¬ 
dard  deviation  of  the  error  for  each  indenter  and  dis¬ 
placement. 

The  accuracy  of  the  sensor  degraded  as  the  deformation 


Displacement 

(mm) 

Mean  error 
(mm) 

Std.  deviation 

1 

0.1806 

0.2486 

2 

0.3895 

0.3099 

3 

0.8228 

0.4091 

4 

1.5930 

0.6613 

Table  1:  Point  indenter. 


Displacement 

(mm) 

Mean  error 
(mm) 

Std.  deviation 

1 

0.2444 

0.2472 

2 

0.5356 

0.4074 

3 

0.9489 

0.6094 

4 

1.4700 

0.4994 

Table  2:  Flat  indenter. 


depth  increased.  This  was  to  be  expected  because  large 
membrane  deformations  violate  the  assumptions  of  the 
reconstruction  algorithm.  The  above  errors  can  be  used 
to  calibrate  subsequent  sensor  measurements. 

Estimation  of  Local  Curvature:  Using  three 
different  indenters,  a  deformation  of  4?nm  was  applied 
along  the  surface  normal,  near  the  center  of  the  fin- 
gerpad  and  the  reconstructed  grid  was  obtained.  This 
process  was  repeated  twenty  times.  The  three  inden¬ 
ters  used  were  the  “point”  and  flat  indenters  described 
above,  as  well  as  a  1.27cm-diameter  sphere.  For  each 
reconstructed  grid,  we  identified  the  membrane  loca¬ 
tion  that  was  maximally  displaced.  At  that  point  of 
maximum  displacement,  we  numerically  computed  the 
rate  of  change  of  the  surface  tangent  along  two  vectors 
that  formed  a  local  basis  for  the  surface.  The  tangents’ 
rate  of  change  provided  an  estimate  of  local  curvature 
at  the  contact.  Table  3  shows  the  mean  and  standard 
deviation  for  each  group  of  estimates.  These  means  are 


Indenter 

Mean  (cm  1 ) 

Std.  Deviation 

Point 

0.53 

0.01 

Ball 

0.47 

0.02 

Flat 

0.20 

0.01 

Table  3:  Curvature  measurements. 


to  be  compared  with  the  actual  curvatures  of  the  in¬ 
denters  which  were  0  for  the  flat  disk,  0.79cm_1  for  the 
ball  and  3.94cm.-1  for  the  point  indenter.  The  com¬ 
parison  shows  poor  agreement,  due  to  the  fact  that 
the  reconstructed  grid  was  twice  differentiated  numer¬ 
ically  and  also  because  significant  membrane  deforma¬ 
tion  was  required  to  effect  a  large  enough  contact  area 
from  which  curvature  information  could  be  extracted. 
Despite  their  low  accuracy,  the  estimates  in  Table  3  do 
exhibit  the  right  trend  and  can  be  used  to  distinguish 
between  different  curvatures,  albeit  with  rather  coarse 
quantization. 


The  accuracy  of  the  curvature  measurements  can  be 
improved  by  sampling  the  membrane  surface  more 
densely  (drawing  and  imaging  a  denser  grid  of  dots). 
It  should  be  noted  that  the  curvature  estimates  for 
the  point  and  ball  indenters  are  comparable  partly  be¬ 
cause  the  membrane  cannot  deform  perfectly  around 
the  point  indenter,  making  it  indistinguishable  from  a 
variety  of  slightly  larger  conical  indenters. 


5  Conclusions  and  Future  Work 

We  have  presented  the  results  of  tactile  sensing  experi¬ 
ments  with  a  new,  deformable,  gel-filled  tactile  sensor. 
Previous  work  (Shimoga  &  Goldenberg)  has  shown  the 
superiority  of  gel  fingertips  for  force  dissipation  and 
conformability.  Our  sensor  reconstructs  the  shape  of 
an  elastic  membrane  using  image  data,  thus  providing 
a  rich  set  of  tactile  information.  The  experiments  pre¬ 
sented  here  demonstrated  the  performance  of  this  sen¬ 
sor  in  simple  tasks  involving  contact  localization,  spa¬ 
tial  resolution,  contact  depth  and  curvature  discrimi¬ 
nation.  For  small  deformations  of  the  membrane,  the 
contact  localization  error  was  less  than  0.1mm  over  a 
4cm2  area,  while  the  spatial  resolution  was  better  than 
2mm.  The  sensor  can  accurately  determine  deforma¬ 
tion  depth  for  small  deformations.  Curvature  estima¬ 
tion  is  monotonic,  however  the  estimates  suffer  from 
the  low  resolution  of  the  numerical  data  used  to  com¬ 
pute  derivatives. 

The  experiments  discussed  here  involved  geometrically- 
defined  idealized  tasks.  Results  on  the  use  of  our  sen¬ 
sor  in  manipulation  experiments  are  presented  in  [16]. 
Other  applications  being  explored  include  the  minia¬ 
turization  of  the  sensor  and  use  as  a  laparoscopic  de¬ 
vice  in  minimally-invasive  surgery.  Future  work  may 
include  the  inclusion  of  a  load  cell  within  the  sealed 
membrane  in  order  to  better  estimate  membrane  and 
contact  forces. 
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